<html>
<head><meta charset="utf-8"><title>rust-analyzer vfs · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html">rust-analyzer vfs</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="173691947"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173691947" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173691947">(Aug 20 2019 at 20:14)</a>:</h4>
<p>Hello all!</p>
<p>I've been working on a project with its own VFS implementation that looks really similar to what ra_vfs is doing. Is this kind of crate a good candidate for generalizing for the rest of the Rust ecosystem?</p>
<p>My project's VFS strategy was pretty similar to ra_vfs, with tracking of multiple roots and reading in contents pretty aggressively. I've been rewriting it recently to ditch the notion of roots and also do I/O lazily on-demand, which obviates the need for ignores in my application and also reduces incidents of my application trying to read files while they're still being saved.</p>
<p>If our use cases line up, I'm interested in factoring out a VFS crate with all the combined wisdom!</p>



<a name="173693760"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173693760" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173693760">(Aug 20 2019 at 20:36)</a>:</h4>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> that's interesting! What do you use for watching files for changes?</p>



<a name="173693842"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173693842" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173693842">(Aug 20 2019 at 20:36)</a>:</h4>
<p>notify 4.x, considering porting over to 5.x since it looks like it uses crossbeam-channel now. Not being able to <code>select</code> over std mpsc channels is troublesome.</p>



<a name="173693965"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173693965" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173693965">(Aug 20 2019 at 20:38)</a>:</h4>
<p>Yeah, so the file watching bit is the thing that interests me most, as it turns out to be the most complicated thing. Issues like this one are a pretty common occurence: <a href="https://github.com/passcod/notify/issues/208" target="_blank" title="https://github.com/passcod/notify/issues/208">https://github.com/passcod/notify/issues/208</a></p>



<a name="173694020"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694020" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694020">(Aug 20 2019 at 20:38)</a>:</h4>
<p>Yep, we get bug reports like that from MacOS users too. D:</p>



<a name="173694068"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694068" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694068">(Aug 20 2019 at 20:39)</a>:</h4>
<p>I'd love to be able to abstract that part. Specifically, having a library on top of notify to which I can tell "watch this direcotry", and which will manage initial walkdir + managing notfications transparently. Like, I want walkdir, which also does watching</p>



<a name="173694141"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694141" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694141">(Aug 20 2019 at 20:40)</a>:</h4>
<p>It'd be cool to extract this behind the API, as I am thinking that maybe using facebook's watchman over notify is a better thing to do...</p>



<a name="173694177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694177">(Aug 20 2019 at 20:40)</a>:</h4>
<p>Re lazy vs. eager file loading, that's an interesting design question! Rust-analyzer is eager by design, but I am not entirely sure that it's the right design....</p>



<a name="173694283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694283">(Aug 20 2019 at 20:42)</a>:</h4>
<p>The hard invariant I want to maintain is that no IO happens during analysis. With eager file loading, I do all the IO upfront, handle all the errors, and then the rest of rust-analyzer's compiler literally never has to deal with errors.</p>



<a name="173694315"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694315" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694315">(Aug 20 2019 at 20:43)</a>:</h4>
<p>I wonder if it is possible to marry lazy loading and absence of errors though.... Like, on the error you can return an empty file, and just signal the error via a side channel</p>



<a name="173694340"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694340" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694340">(Aug 20 2019 at 20:43)</a>:</h4>
<p>That was an invariant we had for ours too. The base of the application is basically webpack, so user-configurable transformation of files into whatever else, and we model those as pure functional snapshot functions in terms of the in-memory files</p>



<a name="173694341"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694341" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694341">(Aug 20 2019 at 20:43)</a>:</h4>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> is your code published somewhere? I am definitelly interested in at least taking a look.</p>
<p>Also, rls 1.0 also has a vfs, which is lazy, so that might be interesting, code-sharing-wise, as well</p>



<a name="173694460"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694460" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694460">(Aug 20 2019 at 20:45)</a>:</h4>
<p>Another reason why we want eager loading is to be abdle to crawl all files in the current package, to be able to index symbols for "go to symbol" even if a particular file is not included into any crate as a module</p>



<a name="173694477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694477">(Aug 20 2019 at 20:45)</a>:</h4>
<p>But, presumably, directory walking can be implemented on top of lazy loading as well?</p>



<a name="173694551"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694551" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694551">(Aug 20 2019 at 20:46)</a>:</h4>
<p>Yep! New implementation tracks for directories whether the children have been enumerated yet, and files just store an <code>Option&lt;Vec&lt;u8&gt;&gt;</code> depending on whether the contents have been loaded.</p>



<a name="173694625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694625">(Aug 20 2019 at 20:47)</a>:</h4>
<p>Aha, interesting! And how do you setup watching, if you don't have explicit roots? Something like "track which directories/files are actually accessed, and watch those"?</p>



<a name="173694694"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694694" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694694">(Aug 20 2019 at 20:48)</a>:</h4>
<p>Yeah, that's the current chunk I'm working on this week</p>



<a name="173694758"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173694758" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173694758">(Aug 20 2019 at 20:48)</a>:</h4>
<p>We were having issues where users had their Git indexes get read into the VFS, but we didn't want to explicitly list ignores as globs or whatever</p>



<a name="173695055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695055">(Aug 20 2019 at 20:52)</a>:</h4>
<p>Aha, that makes sense, ignores are a pain to get right and convenient to use. </p>
<p>How your VFS incorporates changes? In rust-analyzer, we have this elaborate <code>VfsTask</code> things, which allow us to separate noticing that something's changed from actually applying the changes, which makes changes transactional. I like the property that vfs can't change under your feet (which is not the case with rls's vfs), but I wonder if this can be achieved easier...</p>



<a name="173695072"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695072" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695072">(Aug 20 2019 at 20:52)</a>:</h4>
<p>The (pretty sloppy, oof) implementation we ship right now hangs out here:<br>
<a href="https://github.com/rojo-rbx/rojo/blob/6979f5c82d08a72a9e4bc0fc9d30383d799440e6/server/src/imfs.rs" target="_blank" title="https://github.com/rojo-rbx/rojo/blob/6979f5c82d08a72a9e4bc0fc9d30383d799440e6/server/src/imfs.rs">https://github.com/rojo-rbx/rojo/blob/6979f5c82d08a72a9e4bc0fc9d30383d799440e6/server/src/imfs.rs</a></p>
<p>It tracks roots like ra_vfs does, with the file watching/application-specific change tracking rolled into one spot here:<br>
<a href="https://github.com/rojo-rbx/rojo/blob/6979f5c82d08a72a9e4bc0fc9d30383d799440e6/server/src/fs_watcher.rs" target="_blank" title="https://github.com/rojo-rbx/rojo/blob/6979f5c82d08a72a9e4bc0fc9d30383d799440e6/server/src/fs_watcher.rs">https://github.com/rojo-rbx/rojo/blob/6979f5c82d08a72a9e4bc0fc9d30383d799440e6/server/src/fs_watcher.rs</a></p>
<p>Current work that is definitely _not_ ready is in this part of this branch:<br>
<a href="https://github.com/rojo-rbx/rojo/tree/reconciler-mk5/server/src/imfs" target="_blank" title="https://github.com/rojo-rbx/rojo/tree/reconciler-mk5/server/src/imfs">https://github.com/rojo-rbx/rojo/tree/reconciler-mk5/server/src/imfs</a></p>
<p>I want to steal the integer file handle idea from ra_vfs, since we just clone <code>PathBuf</code> objects everywhere right now (ahh!), considering using something like <code>generational-arena</code> to get a little better guarantees on top of that.</p>
<p>We've also got an abstraction of a hierarchical path map that underpins this. It's alright, but a little funky:<br>
<a href="https://github.com/rojo-rbx/rojo/blob/reconciler-mk5/server/src/path_map.rs" target="_blank" title="https://github.com/rojo-rbx/rojo/blob/reconciler-mk5/server/src/path_map.rs">https://github.com/rojo-rbx/rojo/blob/reconciler-mk5/server/src/path_map.rs</a></p>



<a name="173695128"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695128" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695128">(Aug 20 2019 at 20:53)</a>:</h4>
<p>sweet, reading all that right now!</p>



<a name="173695131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695131">(Aug 20 2019 at 20:53)</a>:</h4>
<p>We apply them as the changes happen, everything is held by a top-level mutex since file changes always send us into a big snapshot function that wants exclusive access.</p>



<a name="173695313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695313">(Aug 20 2019 at 20:55)</a>:</h4>
<p>Dealing with diffing+applying changes is something that's core to this project, Rojo, as well, since we turn file change events into snapshots, then turn those into diff objects that we apply to a DOM, then send over the wire to another application via IPC. :D</p>



<a name="173695464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695464">(Aug 20 2019 at 20:57)</a>:</h4>
<p>Oh, the other motivation for lazy I/O for us is that we don't know what the roots of the project are until we explore the project, since we want to support nested project manifests automagically.</p>



<a name="173695512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695512">(Aug 20 2019 at 20:57)</a>:</h4>
<p>Yeah, I feel like your's vfs is father along than rust-analyzer's</p>



<a name="173695622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695622">(Aug 20 2019 at 20:58)</a>:</h4>
<p>Do you deal with symlinks already?</p>



<a name="173695679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695679">(Aug 20 2019 at 20:59)</a>:</h4>
<p>A significant motivation behind the roots system was to make sure that vfs's file system is a real tree, and not an arbitrary graph like the real file system is.</p>



<a name="173695832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695832">(Aug 20 2019 at 21:01)</a>:</h4>
<p>We don't do anything explicitly with symlinks right now, we might even have duplicate file contents bugs? I'm sure we crash for cyclical symlinks right now</p>



<a name="173695900"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173695900" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173695900">(Aug 20 2019 at 21:02)</a>:</h4>
<p>I haven't had good luck with path canonicalization, especially when racing with other applications deleting files and causing it to fail.</p>



<a name="173696126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696126">(Aug 20 2019 at 21:05)</a>:</h4>
<p>Yeah, that's kind of a tough design choice:</p>
<p>For "cli" apps, the wisdom says that you should never change paths you get from user/os.</p>
<p>For something like rust-analyzer though, I'd love to be able to "walk the subdiretory tree", without feaing of infinite loops and such, even if that means showing an error to the user with "sorry, you have an extremely weird file system setup, can't handle it fully"</p>



<a name="173696211"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696211" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696211">(Aug 20 2019 at 21:06)</a>:</h4>
<p>Does notify handle those cases gracefully today?</p>



<a name="173696253"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696253" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696253">(Aug 20 2019 at 21:06)</a>:</h4>
<p>I have no idea, but I think nothing in notify itself walks the directories recursively, so it's not their problem</p>



<a name="173696365"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696365" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696365">(Aug 20 2019 at 21:08)</a>:</h4>
<p>It does! The second arg to <code>Watcher::watch</code> has a <code>RecursiveMode</code> argument, we use it and it seems to work alright: <a href="https://docs.rs/notify/4.0.12/notify/trait.Watcher.html#tymethod.watch" target="_blank" title="https://docs.rs/notify/4.0.12/notify/trait.Watcher.html#tymethod.watch">https://docs.rs/notify/4.0.12/notify/trait.Watcher.html#tymethod.watch</a></p>



<a name="173696379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696379">(Aug 20 2019 at 21:08)</a>:</h4>
<p>I wonder if there's a middle ground here...</p>
<p>I imagine a good thing to have would be "as true to OS as possible" VFS, which doesn't have roots and uses <code>std::Path</code> everywhere. On top of this thing, I might build a roots system with <code>RelativePath</code> paths. I need to think If I really need roots though... But it seems I might: "walk all .rs files in this dir" is something that an IDE needs</p>



<a name="173696411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696411">(Aug 20 2019 at 21:09)</a>:</h4>
<p>We have a handful of bugs about dropped changes but I don't know whether they get dropped by the OS, by notify, or by our code, and it's a crapshoot trying to debug them.</p>



<a name="173696663"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696663" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696663">(Aug 20 2019 at 21:12)</a>:</h4>
<p>That is the code that incorporates changes, right? </p>
<p><a href="https://github.com/rojo-rbx/rojo/blob/e45dd9fc0021e6f57baff093a7eb335290642d5e/server/src/fs_watcher.rs#L85-L143" target="_blank" title="https://github.com/rojo-rbx/rojo/blob/e45dd9fc0021e6f57baff093a7eb335290642d5e/server/src/fs_watcher.rs#L85-L143">https://github.com/rojo-rbx/rojo/blob/e45dd9fc0021e6f57baff093a7eb335290642d5e/server/src/fs_watcher.rs#L85-L143</a></p>



<a name="173696698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696698">(Aug 20 2019 at 21:13)</a>:</h4>
<p>That's what ties notify to the VFS and our project "session", yeah</p>



<a name="173696749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696749">(Aug 20 2019 at 21:14)</a>:</h4>
<p>We still crawl all kinds of files in the project directories, so that doesn't feel tied to roots to me</p>



<a name="173696760"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696760" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696760">(Aug 20 2019 at 21:14)</a>:</h4>
<p>Is there API to get "the list of changed files since I've last looked"?</p>



<a name="173696888"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173696888" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173696888">(Aug 20 2019 at 21:15)</a>:</h4>
<p>Not in <code>master</code>, but I want to start using an unbounded channel for that in <code>reconciler-mk5</code> with the new vfs</p>



<a name="173697012"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697012" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697012">(Aug 20 2019 at 21:17)</a>:</h4>
<p>Cool! That would be crucial for rust-analyzer, because we need to eagarly invalidate the "caches".  So far, I must say I extremely like everything about your approach besides the eager update via locked mutex!</p>



<a name="173697120"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697120" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697120">(Aug 20 2019 at 21:18)</a>:</h4>
<blockquote>
<p>unbounded channel f</p>
</blockquote>
<p>I am not sure I understand, but "unbounded channel" sounds like a think which can make coalescing changes harder. LIke, if the user saves files ten times, ideally I would prefer to get only one changed event</p>



<a name="173697278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697278">(Aug 20 2019 at 21:21)</a>:</h4>
<p>This is something I haven't thought through enough to implement yet. The idea with using an unbounded queue would be that you'd <code>try_recv</code> it (without blocking) until it's empty, merge the events together, and then process them.</p>



<a name="173697340"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697340" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697340">(Aug 20 2019 at 21:22)</a>:</h4>
<p>I guess you could do that on the side before the changes are sent, but that feels like it's getting into the same problems that notify's debounced event impl has</p>



<a name="173697450"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697450" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697450">(Aug 20 2019 at 21:24)</a>:</h4>
<p>In <code>ra_vfs</code>, we have an explicit <code>commit</code> method that advances the version and gives you the list of changed files. It doens't do coalescing though, but it should be able to rather trivially:</p>
<p><a href="https://github.com/rust-analyzer/ra_vfs/blob/bdd92a19231dcf556e048322ebac7af6701d52e0/src/lib.rs#L257-L261" target="_blank" title="https://github.com/rust-analyzer/ra_vfs/blob/bdd92a19231dcf556e048322ebac7af6701d52e0/src/lib.rs#L257-L261">https://github.com/rust-analyzer/ra_vfs/blob/bdd92a19231dcf556e048322ebac7af6701d52e0/src/lib.rs#L257-L261</a></p>



<a name="173697574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697574">(Aug 20 2019 at 21:26)</a>:</h4>
<p>Recalled why we don't use recursive mode of notify, and do it ourselves: <a href="https://github.com/rust-analyzer/rust-analyzer/pull/556#issuecomment-455821185" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/pull/556#issuecomment-455821185">https://github.com/rust-analyzer/rust-analyzer/pull/556#issuecomment-455821185</a></p>



<a name="173697583"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697583" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697583">(Aug 20 2019 at 21:26)</a>:</h4>
<p>with recursive, you can't ignore stuff</p>



<a name="173697615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697615">(Aug 20 2019 at 21:27)</a>:</h4>
<p>Ahhh, I wonder why we haven't run into any significant bug reports based on that same issue.</p>



<a name="173697771"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697771" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697771">(Aug 20 2019 at 21:29)</a>:</h4>
<p>So, I got to go, but I am extremely interested in all this! I think I'll try to whip up an API for vfs that rust-analyzer needs, trying to make it rootless and lazy, and then it makes sense to compare the two. </p>
<p>It would be cool if you can take a closer look at the <code>commit</code> thing: I feel whole-fs snapshottng is pretty crucial for ra, and, from the looks of it, imfs doesn't provide it (just like the real fs doesn't provide it).</p>



<a name="173697889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173697889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173697889">(Aug 20 2019 at 21:30)</a>:</h4>
<p>Definitely, I think that approach is also super neat from a robustness perspective.</p>



<a name="173755515"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173755515" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173755515">(Aug 21 2019 at 00:43)</a>:</h4>
<p>The join-on-drop <code>ScopedThread</code> type is neat, that feels like it should be a crate since Crossbeam's scoped thread is for a different purpose (borrowing things from your stack)</p>



<a name="173756108"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173756108" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173756108">(Aug 21 2019 at 00:58)</a>:</h4>
<p>In ra_vfs, who is doing work with the changes returned from <code>commit</code>? In imfs, it's the same thread that receives the notify change through a channel that ultimately does I/O and updates the imfs.</p>



<a name="173756209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173756209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jane Lusby <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173756209">(Aug 21 2019 at 01:00)</a>:</h4>
<p>uh oh, join on drop scoped thread. sounds like the leakpocalypse again</p>



<a name="173756277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173756277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173756277">(Aug 21 2019 at 01:02)</a>:</h4>
<p>luckily it's still <em>safe</em> to leak it since it doesn't hold stack references! <span aria-label="big smile" class="emoji emoji-1f604" role="img" title="big smile">:big_smile:</span></p>



<a name="173757651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173757651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jane Lusby <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173757651">(Aug 21 2019 at 01:41)</a>:</h4>
<p>oooo got it got it got it. That kind of scoped thread, not the other kind of scoped thread, that's what you meant earlier. All making sense now</p>



<a name="173770586"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173770586" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> ylck <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173770586">(Aug 21 2019 at 07:21)</a>:</h4>
<p>cool</p>



<a name="173771133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173771133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173771133">(Aug 21 2019 at 07:33)</a>:</h4>
<p>published scoped thread as <a href="https://crates.io/crates/jod-thread" target="_blank" title="https://crates.io/crates/jod-thread">https://crates.io/crates/jod-thread</a></p>



<a name="173773339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173773339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173773339">(Aug 21 2019 at 08:11)</a>:</h4>
<p><span class="user-mention" data-user-id="220273">@Jane Lusby</span> yeah, the purpose of <code>jod-thread</code> is not to express more lifetime patterns, but to make software more robust. Not joining a thread is bad because, for example, you might get interference between zombie threads of various tests. Additionally, if you don't join a thread and exit the <code>main</code>, then thread's destructors won't run</p>



<a name="173773955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173773955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173773955">(Aug 21 2019 at 08:22)</a>:</h4>
<blockquote>
<p>In ra_vfs, who is doing work with the changes returned from commit? In imfs, it's the same thread that receives the notify change through a channel that ultimately does I/O and updates the imfs.</p>
</blockquote>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> that's a little wonky at the moment. For IO specifically, I try (but not 100% succesfully) to maintain constraint than only one thread does IO. This helps to make sure that the history of file versions always goes forward in time: if one thread eagarly loads the file while crawling a directory, and nother thread eagarly loads a file due to notification from notify, they can race against each other and you might end up with <em>older</em> file contents in the VFS. I think that with lazy loading you don't have this problem. </p>
<p>Incorporation of the new file content, read from disk, into the VSF happens in the app's main event loop.</p>
<p>Specifically, here we get a bunch of files from IO thread and add them to vfs as "pending changes": <a href="https://github.com/rust-analyzer/rust-analyzer/blob/bdf16d1b670959f35bcbadbfd11b5c1b5a396703/crates/ra_lsp_server/src/main_loop.rs#L227-L229" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/bdf16d1b670959f35bcbadbfd11b5c1b5a396703/crates/ra_lsp_server/src/main_loop.rs#L227-L229">https://github.com/rust-analyzer/rust-analyzer/blob/bdf16d1b670959f35bcbadbfd11b5c1b5a396703/crates/ra_lsp_server/src/main_loop.rs#L227-L229</a>.</p>
<p>After that, we commit those pending changes and get the new state of vfs: <a href="https://github.com/rust-analyzer/rust-analyzer/blob/bdf16d1b670959f35bcbadbfd11b5c1b5a396703/crates/ra_lsp_server/src/main_loop.rs#L260" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/bdf16d1b670959f35bcbadbfd11b5c1b5a396703/crates/ra_lsp_server/src/main_loop.rs#L260">https://github.com/rust-analyzer/rust-analyzer/blob/bdf16d1b670959f35bcbadbfd11b5c1b5a396703/crates/ra_lsp_server/src/main_loop.rs#L260</a></p>
<p>The problem here is that we don't actually really maintain a list of pending changes, rather, we update in-place but record what was updated, and then the <code>commit</code> message only returns what has changed, while ideally it also should be the method that applies the changes.</p>



<a name="173776770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173776770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173776770">(Aug 21 2019 at 09:11)</a>:</h4>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> so, I think this is roughly the API I want to see: <a href="https://gist.github.com/matklad/edd119db4434774695d30491dc5cb428" target="_blank" title="https://gist.github.com/matklad/edd119db4434774695d30491dc5cb428">https://gist.github.com/matklad/edd119db4434774695d30491dc5cb428</a></p>



<a name="173776833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173776833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173776833">(Aug 21 2019 at 09:12)</a>:</h4>
<p>Just realized that <span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> might also be interested in this ^^ VFS discussion :D</p>



<a name="173811292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173811292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173811292">(Aug 21 2019 at 16:40)</a>:</h4>
<p>Our project just made a similar transition for change tracking in the DOM diffing phase. We used to apply changes and log what changed as it happened, but now we generate + apply patch objects!</p>



<a name="173816152"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173816152" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173816152">(Aug 21 2019 at 17:42)</a>:</h4>
<p>That API looks very doable, especially if you're a little flexible on <code>&amp;self</code> vs <code>&amp;mut self</code></p>
<p>Another use we have for our VFS is dropping file change notifications that come from our own program writing the file, to avoid us getting in a nasty loop.</p>



<a name="173816407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173816407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173816407">(Aug 21 2019 at 17:46)</a>:</h4>
<p>You write to files via VFS as well, right? It's interesting that in rust-analyzer we don't have this problem: when we want to write to a file, we ask the editor to do this, and then we see a change via usual vfs notification mechanism</p>



<a name="173816425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173816425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173816425">(Aug 21 2019 at 17:46)</a>:</h4>
<p>That is, we don't directly mutate VFS</p>



<a name="173816492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173816492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173816492">(Aug 21 2019 at 17:47)</a>:</h4>
<blockquote>
<p>That API looks very doable, especially if you're a little flexible on <code>&amp;self</code> vs <code>&amp;mut self</code></p>
</blockquote>
<p>That depends on what "flexible" mean. I use <code>&amp;</code> and <code>&amp;mut</code> to enforce consistent shapshots, but I am fine with any other way of enforcement</p>



<a name="173816853"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173816853" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173816853">(Aug 21 2019 at 17:51)</a>:</h4>
<p>With lazy I/O, <code>VFile::contents</code>, <code>VDir::list</code> and <code>VDir::walk</code> would probably need to accept <code>&amp;mut Vfs</code> since they can change the state of the VFS. Do you think that's at odds with consistent snapshots, since it opens up a potential for a race?</p>



<a name="173816917"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173816917" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173816917">(Aug 21 2019 at 17:51)</a>:</h4>
<p>We write to files through the VFS, yeah, and then when we get a change notification we can compare the new contents with our existing and drop the change on the ground</p>



<a name="173817121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173817121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173817121">(Aug 21 2019 at 17:54)</a>:</h4>
<p>Hehe, I bet <code>&amp;self</code> would work, and would guantee consistent snapshots...</p>
<p>Let me draft a little bit of code...</p>



<a name="173817844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173817844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173817844">(Aug 21 2019 at 18:02)</a>:</h4>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> so, something like this is possible:</p>
<div class="codehilite"><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">fs</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">path</span>::<span class="n">PathBuf</span><span class="p">;</span><span class="w"></span>

<span class="k">use</span><span class="w"> </span><span class="n">once_cell</span>::<span class="n">sync</span>::<span class="n">OnceCell</span><span class="p">;</span><span class="w"></span>

<span class="k">struct</span> <span class="nc">FileData</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">path</span>: <span class="nc">PathBuf</span><span class="p">,</span><span class="w"></span>
<span class="w">    </span><span class="n">contents</span>: <span class="nc">OnceCell</span><span class="o">&lt;</span><span class="nb">Option</span><span class="o">&lt;</span><span class="nb">Vec</span><span class="o">&lt;</span><span class="kt">u8</span><span class="o">&gt;&gt;&gt;</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="k">impl</span><span class="w"> </span><span class="n">FileData</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">reset</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="bp">self</span><span class="p">.</span><span class="n">contents</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">OnceCell</span>::<span class="n">new</span><span class="p">()</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>

<span class="w">    </span><span class="k">fn</span> <span class="nf">get</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nb">Option</span><span class="o">&lt;&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">]</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="bp">self</span><span class="p">.</span><span class="n">contents</span><span class="w"></span>
<span class="w">            </span><span class="p">.</span><span class="n">get_or_init</span><span class="p">(</span><span class="o">||</span><span class="w"> </span><span class="n">fs</span>::<span class="n">read</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">as_path</span><span class="p">()).</span><span class="n">ok</span><span class="p">())</span><span class="w"></span>
<span class="w">            </span><span class="p">.</span><span class="n">as_ref</span><span class="p">()</span><span class="w"></span>
<span class="w">            </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">it</span><span class="o">|</span><span class="w"> </span><span class="n">it</span><span class="p">.</span><span class="n">as_slice</span><span class="p">())</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>



<a name="173817909"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173817909" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173817909">(Aug 21 2019 at 18:03)</a>:</h4>
<p>That's file you can lazy read with <code>&amp;</code> and reset with <code>&amp;mut</code>. I am not sure if it transers easitly to hash map though, but I imagine it should</p>



<a name="173818681"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173818681" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173818681">(Aug 21 2019 at 18:12)</a>:</h4>
<p>That's a sweet way to encode that relationship for files, nice. It might not work for directory listing, which needs mutable access to the whole VFS to be lazy, though <span aria-label="exhausted" class="emoji emoji-1f625" role="img" title="exhausted">:exhausted:</span></p>



<a name="173819226"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173819226" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Mcnab <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173819226">(Aug 21 2019 at 18:19)</a>:</h4>
<p>Can we learn from Noria's approach? That is maintaining two maps, one read only and one being written to, then swapping them, resetting the writing one once the final reader has been dropped</p>



<a name="173819281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173819281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Mcnab <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173819281">(Aug 21 2019 at 18:19)</a>:</h4>
<p>Of course I can't find the actual code for that - I just remember watching the talk.</p>



<a name="173819681"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173819681" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173819681">(Aug 21 2019 at 18:24)</a>:</h4>
<p>ok, i've spend 10 minutes looking for a just right arena crate, and haven't found one, but I think dirs should be doable in theory as well...</p>



<a name="173819749"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173819749" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173819749">(Aug 21 2019 at 18:25)</a>:</h4>
<p>What we need is an arena with <code>fn push(&amp;self /* sic! */, T) -&gt; usize</code>, <code>fn get(&amp;self, usize) -&gt; &amp;T</code> and <code>fn get_mut(&amp;mut self, usize) -&gt; &amp;mut T</code> methods I think</p>



<a name="173819857"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173819857" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173819857">(Aug 21 2019 at 18:26)</a>:</h4>
<p>that is, we basically allocate files as never-moved slots</p>



<a name="173820114"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173820114" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173820114">(Aug 21 2019 at 18:29)</a>:</h4>
<p>Hm, even</p>
<div class="codehilite"><pre><span></span>enum Entry { Dir(Dir), File(File)}

struct Dir { entries: OnceCell&lt;HashMap&lt;String, Entry&gt;&gt; }

struct File { contents: OnceCell&lt;Vec&lt;u8&gt;&gt;}
</pre></div>


<p>works, though you don't get a flat hash-map view of the file-system....</p>



<a name="173821440"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173821440" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173821440">(Aug 21 2019 at 18:45)</a>:</h4>
<p>maybe I am overthinking this, and a single RWLock around the vfs would be enough? I definitelly want the ability to use VFS from diffrent threads though...</p>



<a name="173826727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173826727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173826727">(Aug 21 2019 at 19:52)</a>:</h4>
<p>It seems like a trade-off between lazy I/O and concurrent read-only access from multiple threads</p>



<a name="173826775"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173826775" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173826775">(Aug 21 2019 at 19:53)</a>:</h4>
<p>If you go down the road of needing to break your path into pieces and pointer-hop through nested hashmaps, that approach works, but I bounced off that idea awhile ago.</p>



<a name="173843790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173843790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173843790">(Aug 22 2019 at 00:39)</a>:</h4>
<p>I thought I had a proof-of-concept of the file-watching implementation working, but now everything is deadlocking on drop, oof.</p>



<a name="173845385"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173845385" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173845385">(Aug 22 2019 at 01:21)</a>:</h4>
<p>notify is holding onto its <code>mpsc::Sender</code> after the watcher is dropped, which is causing the receiver half to never return, lovely!<br>
Switching from <code>RecommendedWatcher</code> to <code>NullWatcher</code> makes the deadlock go away with no other changes.</p>



<a name="173857127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857127">(Aug 22 2019 at 06:46)</a>:</h4>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> , I am pretty sure that in case of ra_fvs, dropping the <code>notify::watcher</code>closes the notify channel. But I did get a deadlock around that area with <em>my</em> channel though.</p>



<a name="173857206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857206">(Aug 22 2019 at 06:48)</a>:</h4>
<p>To ensure that the thread and channels are closed in the right order, I had to use a qute, aweful trick:</p>
<div class="codehilite"><pre><span></span>let thread; // &lt;- affects drop order
let (tx, rx) = channel();
thread = spawn(|| /*use rx here*/);
</pre></div>


<p><a href="https://github.com/rust-analyzer/ra_vfs/blob/af1a6ace3d0edf57d62a76321e3e52eeb99d6d4c/src/io.rs#L78-L83" target="_blank" title="https://github.com/rust-analyzer/ra_vfs/blob/af1a6ace3d0edf57d62a76321e3e52eeb99d6d4c/src/io.rs#L78-L83">https://github.com/rust-analyzer/ra_vfs/blob/af1a6ace3d0edf57d62a76321e3e52eeb99d6d4c/src/io.rs#L78-L83</a></p>



<a name="173857312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857312">(Aug 22 2019 at 06:50)</a>:</h4>
<p>I'm ensuring drop order with struct order, and also tried doing the same thing with <code>ManuallyDrop</code> -- I drop the notify watcher first and then drop the thread, joining it.</p>



<a name="173857325"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857325" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857325">(Aug 22 2019 at 06:50)</a>:</h4>
<p>I'm pretty sure it's a notify bug because it only happens on Windows -- the null watcher and the default on Linux do not deadlock in this way :/</p>



<a name="173857358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857358">(Aug 22 2019 at 06:51)</a>:</h4>
<p>Interesting! I haven't tested this on windows a lot, so it might be the case.</p>



<a name="173857502"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857502" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857502">(Aug 22 2019 at 06:54)</a>:</h4>
<p>(but yeah, I must say that hitting such platform specific bugs in platform-agnostic-ish <code>notify</code> is not pleasant. And I am not sure that it even moves in the right direction, given super grand plans for 5.0. For something where correctness is criticla, I'd seriously consider trying watchman)</p>



<a name="173857581"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857581" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857581">(Aug 22 2019 at 06:56)</a>:</h4>
<p>The behavior around watching things across renames is definitely pretty spooky across different platforms.</p>



<a name="173857596"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857596" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857596">(Aug 22 2019 at 06:57)</a>:</h4>
<p>Is there a watchman bindings crate?</p>



<a name="173857612"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/173857612" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#173857612">(Aug 22 2019 at 06:57)</a>:</h4>
<p>I don't think so. And given that it's an external process, there should be a special can of worms to it as well...</p>



<a name="174321771"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/174321771" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#174321771">(Aug 28 2019 at 00:21)</a>:</h4>
<p>One complication we discovered this week (in our VFS project!) is that our ad-hoc roots can be removed and we need to handle it gracefully, which is starting to complicate watching.</p>



<a name="174321857"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/174321857" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#174321857">(Aug 28 2019 at 00:23)</a>:</h4>
<p>For example, the user can configure their project to watch <code>src/foo</code> and <code>src/bar</code>. We need to handle either of them being removed and re-added, but also need to handle (to some degree) the user deciding that they don't care about either of those folders anymore. It feels almost like we need to reference count our watches, but that's tricky because we aren't tracking roots at the VFS level anymore :(</p>



<a name="174448090"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/174448090" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#174448090">(Aug 29 2019 at 12:23)</a>:</h4>
<p>Interesting! It seems like changing the set of roots is a rare operation, so perhaps a solution is just a linear traversal that takes the current set of roots and the current set of watches and reconsiles them?</p>



<a name="177719289"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177719289" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177719289">(Oct 09 2019 at 14:32)</a>:</h4>
<p><span class="user-mention" data-user-id="222367">@Lucien Greathouse</span> I am starting to move to the new VFS in <a href="https://github.com/matklad/rust-analyzer/commit/a602a5a715e354d633ac90d7ddf47d3176f1e986#diff-429ed516d2631cff68f57516b8644a09" target="_blank" title="https://github.com/matklad/rust-analyzer/commit/a602a5a715e354d633ac90d7ddf47d3176f1e986#diff-429ed516d2631cff68f57516b8644a09">https://github.com/matklad/rust-analyzer/commit/a602a5a715e354d633ac90d7ddf47d3176f1e986#diff-429ed516d2631cff68f57516b8644a09</a></p>



<a name="177736102"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177736102" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177736102">(Oct 09 2019 at 17:22)</a>:</h4>
<p>Nice, this looks good. I haven't moved on our VFS implementation and have focused on other things since it mostly-kinda works.</p>



<a name="177931316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177931316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177931316">(Oct 11 2019 at 17:50)</a>:</h4>
<p>I see that <code>ra_vfs</code> is internally locked with an <code>RwLock</code>, and all the file contents are kept as <code>Arc&lt;String&gt;</code>. I've been struggling pretty badly with synchronization in our implementation, so this looks really sound.</p>



<a name="177931353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177931353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177931353">(Oct 11 2019 at 17:51)</a>:</h4>
<p>Notably, we found that once you start making every file operation lazy, it's way too hard to hold onto data and it increased our data copying a lot. Refcounting the buffers (since they're copy-on-write anyways) is awesome.</p>



<a name="177932590"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177932590" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177932590">(Oct 11 2019 at 18:05)</a>:</h4>
<p>Yeah. Another nice thing I’ve realized is by using this invalidator thing<br>
one can make VFS completely oblivious of he actual watcher implementation.<br>
One not so nice thing that I”ve realized is that I still need roots for<br>
incrementality. Basically, if I notice that a <a href="http://mod.rs" target="_blank" title="http://mod.rs">mod.rs</a> file is created, I<br>
need to invalidate some thing, and, as I don’t want to invalidate<br>
everything, I need a root concept</p>



<a name="177933327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177933327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177933327">(Oct 11 2019 at 18:12)</a>:</h4>
<p>Oh! We modeled that in our "snapshot" subsystem, where for each output artifact we track all paths that could possibly contribute to that artifact.</p>
<p>So we have <code>init.lua</code> as the <code>mod.rs</code> equivalent, and when we snapshot a folder, we say that the relevant paths are <code>foo/</code> and <code>foo/init.lua</code> (along with other stuff, like <code>foo/init.meta.json</code>)</p>



<a name="177934019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177934019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177934019">(Oct 11 2019 at 18:19)</a>:</h4>
<p>Yeah, perhaps I can try to track the precise deps as well? Might be slightly awkward to do with salsa, but should be possible...</p>



<a name="177955297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177955297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lucien Greathouse <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177955297">(Oct 11 2019 at 22:31)</a>:</h4>
<p>Has there been any further investigation in file watching stuff? We're decoupled from Notify now through an interface called <code>ImfsWatcher</code> now. We consume coarser events than what notify gave us. Curious if starting a crate that would start as just a notify wrapper with a smaller API could be worthwhile.</p>



<a name="177999265"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177999265" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177999265">(Oct 12 2019 at 17:48)</a>:</h4>
<p>Sort-of. I took a lot of time to read the docs for watchman. The docs a very thorough when it comes to details, but getting a high-level picture from them seems hard. It seems like it makes sense to model high-level API on top of watchman. Important bits I've got from the docs:</p>
<ul>
<li>watchman also has a concept of root</li>
<li>it manages both scanning and watching: if you subscibe to the dir, it will first emit the "created" event for all the files, so that nothing can be missed. </li>
<li>it supports dynamic registration of new subscriptions. </li>
</ul>
<p>I feel that roughtly the following interface should work for watchers:</p>
<div class="codehilite"><pre><span></span><span class="k">struct</span> <span class="nc">Subscription</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">root</span>: <span class="nc">PathBuf</span><span class="p">,</span><span class="w"></span>
<span class="w">    </span><span class="n">include_glob_pattern</span>: <span class="nb">Option</span><span class="o">&lt;</span><span class="nb">String</span><span class="o">&gt;</span><span class="p">,</span><span class="w"></span>
<span class="w">    </span><span class="n">exclude_glob_pattern</span>: <span class="nb">Option</span><span class="o">&lt;</span><span class="nb">String</span><span class="o">&gt;</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="k">struct</span> <span class="nc">Watcher</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">     </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">event_callback</span>: <span class="nb">Box</span><span class="o">&lt;</span><span class="n">dyn</span><span class="w"> </span><span class="nb">Fn</span><span class="p">(</span><span class="o">&amp;</span><span class="n">Path</span><span class="p">)</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">Watcher</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="p">}</span><span class="w"></span>

<span class="w">     </span><span class="k">fn</span> <span class="nf">set_subscriptions</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">subscription</span>: <span class="nb">Vec</span><span class="o">&lt;</span><span class="n">Subscription</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span><span class="p">{...</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>Interesting points:</p>
<ul>
<li>inclusions and exclusions should be globs and not more general <code>Fn(&amp;Path) -&gt; bool</code>: you can't send a rust fn to watchman or lsp client</li>
<li>that watchig API I think is eager in case of watchman, and I feel like it probably needs to be like this. This still can interoperate with a lazy VFS: basically, VFS gets notified about all changes, but only invalidates actually loaded files.</li>
</ul>



<a name="177999656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/177999656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#177999656">(Oct 12 2019 at 17:59)</a>:</h4>
<p>Oh, another high-order bit I've indetifid about separation between VFS and watching per se. At least in rust-analyzer use-cases, it seems like the set of files in VFS and the set of watched files actually should be different things. Specifically, VFS should contain files from <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a>, but we probably don't want to watch them. We only want to watch files inside the project. Now, we <em>could</em> watch everything, but that'll require more resources. </p>
<p>That's why in a design I have in mind you specify watched roots completely orthogonaly to the actual VFS impl</p>



<a name="178000142"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer%20vfs/near/178000142" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/rust-analyzer.20vfs.html#178000142">(Oct 12 2019 at 18:10)</a>:</h4>
<p>Opened an issue on the watchman repo about my struggle with the docs: <a href="https://github.com/facebook/watchman/issues/753" target="_blank" title="https://github.com/facebook/watchman/issues/753">https://github.com/facebook/watchman/issues/753</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>